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Executive Summary 


The purpose of this organizational review, undertaken by MPR Associates, Inc. at the 
request of Houston A+ Challenge (HA+C), is to enable the organization to take stock 
of its contribution to school reform in the Houston area after a decade of work and 
to examine its weaknesses and strengths as it makes plans for the future. To do this, 
MPR staff visited Houston A+ sites, examined relevant documents, and interviewed 
staff, Board members, and clients. We found that HA+C: 

• Has had a recognized and highly positive effect in the Houston-area 
education community. 

• Is known for being responsive, disseminating research, facilitating 
networks, and providing learning opportunities for students, teachers and 
administrators otherwise unavailable to them. 

• Operates according to a current mission and imperatives that are overly 
broad and not aligned. 

• Allows its mission to be defined by its activities rather than vice versa. 

• Has acquired a large, diverse portfolio of programs and initiatives that can 
be difficult to sustain. 

• Does not have a strong accountability system. 

• Does not communicate its purposes or activities well, both internally and 
externally. 

To address these issues and position the organization for the future, we suggest that 
HA+C: 

• Redefine its mission and focus to reflect its priorities clearly. 

• Develop a strong accountability system and a “culture of evidence” among 
staff and Board. 

• Implement a regular strategic planning process, including the use of 
accountability data, for making funding and programmatic decisions. 

• Consider documenting its work, planning to scale-up successful efforts, and 
terminating unsuccessful ones or those not congruent with the redefined 
mission. 
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• Develop a plan to communicate its redefined mission and priorities to staff, 
Board, clients, funders, and the community at large. 

• Examine the skills and expertise needed by staff and Board members as the 
organization moves ahead. 


Introduction 


In March 2007, Houston A+ Challenge (HA+C) asked MPR Associates (MPR) to 
help assess its “value added” to the progress of school reform in the Houston area. 
MPR reviewed relevant documents and background literature, conducted interviews, 
and submitted a preliminary analysis in April 2007, which confirmed our initial 
sense that HA+C needed a strong accountability system to support strategic planning 
and decision making. A comprehensive and coherent system of accountability could: 

• Provide evidence of success for HA+C as a whole and its programs and 
sponsored initiatives; 

• Become the basis for making strategic decisions to advance the 
organization, and 

• Identify ways to sustain the HA+C core mission. 

Information from such a system could also be invaluable in telling the story of 
HA+C’s work and further establishing the organization as a preeminent educational 
reform support agency in the Houston area. 

At the request of HA+C, MPR offered detailed proposals in May 2007 for evaluating 
1) the organization, 2) the middle school math initiative, and 3) the leadership 
academy. MPR was asked to conduct the first two studies. This report describes only 
the organizational review; the evaluation of the math initiative will be submitted 
separately. Building on our preliminary analysis, we reviewed additional documents 
and data and collected more information from clients, participants, staff, and Board 
members, to assess the overall functioning of HA+C and one of its specific programs. 
Our objective was to create an evidence base for future initiatives and program 
decisions. Information about the methods used to conduct this review is available in 
the Appendix. 

Content of the Report 

We developed this report by analyzing the data we gathered for themes and patterns 
and by conducting debriefing discussions with all research team members. Our 
purpose was not to undertake a comprehensive evaluation of all HA+C’s work, but 
rather to review its operation as a reform support organization — an intermediary — as 
it was established through the original Annenberg Challenge grant. We have 
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organized our findings by specific themes emerging from the data, and we offer 
recommendations for moving the organization forward. The discussion that follows 
focuses on the following topics: 

• HA+C’s positive impact 

• Organizational mission and scope of work 

• Strategic decision making 

• HA+C’s portfolio of programs and initiatives 

• Research and accountability 

• Funding and financial structure 

• Sustaining programs and their impact 

• Maintaining quality control 

• Communication 

• The staff and Board of Directors 
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HA+C’s Positive Impact 


HA+C clearly has had a strong positive effect on the Houston-area educational 
community, according to abundant evidence. In one interview, a principal noted: 
“[They] helped me professionally to have critical conversations in a non-threatening 
way; they helped me with discussing challenges we face and presented me with 
literature that was useful. I purchased books for my staff to do readings together. 
They provide professional development opportunities that [otherwise] I wouldn’t 
have been involved in.” Principals and teachers alike offered specific examples of how 
working with HA+ C had enhanced their professional lives, as in this comment from 
a teacher: “[We] can see beyond HISD to what’s happening regionally and beyond. 
We are expected to read, to develop nationwide awareness — what’s going on in the 
reform world. A+ encourages the collaboration at all levels — as a teacher I didn’t see 
beyond the classroom. In my network, we see principals and assistant principals and 
regional leaders and learn from all of them. Every retreat has been well planned with 
opportunities to reflect on our own practices.” 

HA+C is recognized in the Houston area for being responsive, making cutting-edge 
research available, facilitating networking and educational conversations at many 
levels, and providing learning opportunities for students, teachers, and administrators 
that they otherwise would not have had. Our interviews with participants in a range 
of HA+C-sponsored activities were overwhelmingly positive and laced with many 
examples of HA+C’s influence on their professional lives, their school, and district 
practices. The examples that follow are based on comments made by those interviews 
in discussing a variety of programs in which they participated. The examples reflect 
the fact that the programs share a theoretical approach that involves asking hard 
questions, finding ways to discuss and explore solutions to problems with peers 
within and across districts, and staying abreast of current research on promising 
educational reform strategies. 

• A principal who is a strong proponent of data-based decision-making 
expressed his appreciation for learning about a Critical Friends protocol 
that enabled him to discuss the importance of using data and ways to do so 
with his faculty in a new way. Without this strategy, he would have 
approached this discussion in a much more top-down manner. 
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• A teacher involved in the Critical Friends Group training discussed how 
she applied what she learned in the process to providing instruction in her 
classroom. She used what she learned to redesign her entire classroom 
approach — integrating team-building activities and many more interaction 
and questioning strategies — making her a much better teacher. “I want to 
be a ‘light’ in the district,” she noted. She also emphasized that the work 
had given her an opportunity to work with people all over the country, 
something that wouldn’t have happened without FLA+C. 

• A district administrator currently serving as an interim assistant principal 
said that the opportunity she now has to be in a school allowed her to see 
more clearly “how the idea (being a reflective practitioner) works.” She 
noted that it (the process of facilitating discussions and helping teachers to 
be reflective) was very powerful because at times “our thinking was changed 
by the messages” and that the (reflective) tools provided helped her to 
prioritize and “think about what tools are applicable for the school where 
I’m working.” 

• Finally, a high school teacher who teaches radio/ television communications 
and who participated in an externship conducted interviews with some 
executives at the company where she was placed. She asked about the skills 
they seek in new hires, problems they encounter with new employees, and 
what they wish the schools were teaching students that would be applicable 
in that workplace. This career technical education teacher said she came 
away with valuable information about what employers are looking for and 
what they need, and she learned that students are not able to think 
critically. Company staff confirmed that new hires often do not have strong 
creative thinking skills, which are very valuable to an employer. In her 
classroom, this teacher used what she learned during her externship to push 
her students to think creatively and to find answers themselves, rather than 
simply repeat what they are told. 

These positive effects of participation in FIA+C activities were particularly profound 
for experiences that were long-term and involved a network of participants or 
ongoing support: Critical Friends Group, Leadership Academy, externships, literacy 
initiative. There were, however, also many positive reactions to one-time events that 
brought educational experts to Houston or encouraged sharing expertise at 
conferences. Participants rated their experiences with HA+C very high for quality 
and professionalism, and the examples offered in interviews provide strong evidence 
that many have applied what they have learned in their classrooms, schools, and 
districts. Overall, the broad array and quality of HA+C programs have established it 
as an important school reform agency in the Houston area. 
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Abundant positive evidence from 
HA+C clients confirms a 
significant accomplishment: 
establishing itself as a respected 
school reform organization in an 
urban environment, with a 
continuing and growing client 
base and a history of solid 
funding. 

future. In each case, we have highlighted 

ways to address the issues identified. 


We interviewed only individuals 
involved in one or more HA+C 
activities. Several mentioned that they 
were often surprised, when describing 
their activities to their peers, that 
many did not know anything about 
the organization. We also found that 
some interviewees did not know about 
HA+C programs other than those in 
which they were involved. These 
observations are directly related to the 
theme of communication, as described 
in one of the sections below. The 
sections each discuss themes that we 
consider significant as HA+C maps its 
specific findings and recommendations for 
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Organizational Mission and Scope of Work 

The HA+C website currently states its mission as: 

To promote an academically rich and purposeful education for more of our 
children, and to demonstrate how such an education could become possible for all 
our children. We do this by investing in people, through initiatives that coach 
teachers and school leaders to acquire the skills they need to help students of diverse 
backgrounds succeed academically, professionally and socially. 

HA+C specifies three roles for itself: 1) catalyst, 2) steward of funds, and 3) prompter 
of accountability. It has also operated on three “imperatives” since its beginning: 
personalization, collaboration, and quality learning. This loose configuration of 
beliefs has guided HA+C decisions about which activities or initiatives to support. 
Part of HA+C’s success has been its flexibility as an intermediary and its 
responsiveness to both funders wanting to support improvements in Houston-area 
schools and educators needing resources. 

HA+C’s establishment as an intermediary through which to channel funds and meet 
the Annenberg challenge and to carry out the vision of the multiple organizations in 
the Child-Centered Schools Initiative may explain why it is not governed by a clear, 
focused mission. Now, however, after ten years and numerous projects and programs, 
this historical “soft focus” makes it difficult to articulate HA+C’s central purpose 
clearly. This is a common challenge for intermediary reform support organizations. 1 
But now that much of the work of the original Annenberg Challenge has ended and 
to continue its work in Houston on a similar scale, HA+C must find new resources, 
most likely from a wider variety of funders. The lack of a clearly articulated 
mission — and accomplishments — may make it more difficult for HA+C to secure its 
funding base into the future. 

The organization’s imperatives and mission statement are not truly aligned, and 
neither the mission statement nor the imperatives effectively captures what HA+C 
actually does. The label “imperatives” implies that any investment of the 
organization’s resources must fall within the scope of these activities. Although it is 
easy to see how most ongoing programs address one or more of the imperatives (e.g., 


1 Honig, M., The New Middle Management: Intermediary Organizations in Education Policy 
Implementation (August, 2003). 
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Critical Friends training fosters the creation of school cultures committed to 
collaboration), the connection between the imperatives and the purpose is tenuous, 
and a clear rationale is lacking. Why is it important to promote academic richness? 
Why demonstrate that such an education is possible? How will focusing on the three 
imperatives promote these purposes and to what desired end? 

Rather than a mandate that helps HA+C set priorities, the imperatives are vague 
enough to justify the inclusion of a variety of initiatives. The mission statement, 
then, is derived from HA+C’s activities, rather than serving as a clear statement of 
purpose and guide to how the organization should work. This makes it difficult to 
determine how to scale-up and sustain programs and initiatives and create a financial 
structure to fortify the work. It also makes it difficult to convey the essence of the 
work to potential funders and demonstrate the organization’s impact to those in and 
outside of the education field. 


It can be advantageous for an emerging 
reform organization to have a broad 
mission and flexible scope, because this 
allows a wide variety of clients to 
benefit from its services and helps to 
establish its presence in the reform 
arena. For HA+C, however, the 
consequence of this flexibility has been 
less clarity of purpose. With so much 
on its agenda — individual teacher 
professional development, leadership 
networks, school-level curriculum 
initiatives, high-school reform — even 
those most intimately involved can lose 
sight of the mission. This was clear from the majority of our interviews. Staff 
members knew their particular projects, but had lost sight of an overall purpose and 
reverted most frequently to the “imperatives.” Board members made fun of their own 
inability to “give the elevator speech,” and program participants’ concept of HA+C’s 
overarching purpose was limited to the program(s) in which they were involved. 

As HA+C’s website notes, it invests “in people, rather than programs, by providing 
teachers, principals and administrators with resources, support and coaching to 
improve classroom and district practices.” Measuring and demonstrating the impact 
of such work on students is challenging. Even when respondents stated that the 
ultimate purpose of the organization was student-centered, they often answered the 
question “What about HA+C programs has been most valuable?” by referring to 


Rather than operating from a 
clearly articulated mission that 
determines the goals and work of 
the organization, HA+C has 
allowed external factors to 
determine its programs and 
activities. 
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changes in how teachers and/or administrators learn or interact with one another. 
This shows that the connection between HA+C’s work and improved academic 
opportunities for Houston-area children is obscured. The organization invests in a 
wide variety of activities, but at present has no way to determine which activities are 
most worthwhile. 


HA+C leadership confirmed that they saw a problem with “scope creep,” but the 
alternative that some cited, narrowing the focus to leadership, could also be 
problematic. Among a number of problems one could cite, it could leave many 
clients with unfulfilled expectations regarding programs in which they have 
participated or funds they have received. It could also mean that some willing 
funders find no program or initiative to support that reflects their interest areas. 


Program participants indicated that many see HA+C as a valuable resource for 
helping to launch reform efforts, supporting good instructional programs that may 
be undervalued in a high-stakes accountability policy environment and offering 


HA+ C should more clearly define 
its vision, mission, and focus, but 
the transition should be strategic, 
rather than reactive. 


meaningful professional development 
opportunities. HA+C leadership will 
need to take these values into 
consideration as it develops more 
effective decision-making processes and 
effective impact measures of its impact 
that will help to clarify its mission and 
scope. 


Summary Points and Recommendations 

• HA+C’s broad mission statement and imperatives have enabled it to 
establish a presence and wide visibility in the Houston area. The lack of 
focus in its mission statement, however, has led to a scattered portfolio of 
programs reflecting an organization without a clear purpose. 

• The mission statement is derived from HA+C’s activities, rather than 
serving as an a priori guide to the organization’s purpose and work. 

• HA+C should craft a clear, concise mission statement specifying its 
purpose, to guide strategic planning and enable the organization to be more 
deliberate in its work. 
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Strategic Decision Making 


Because HA+C’s mission is so broad, almost anything called “school reform” could 
be undertaken. As noted by a staff member, “The mission is very broad — we’re going 
to save the world.” This imprecision in HA+C’s purpose has resulted in a scattered 
decision-making process. HA+C should refine its mission to give it a focused 
purpose. 

It appears that program decisions have not been based on a careful examination of 
the trade-offs involved — a consideration of risk and return for specific options. In 
many cases, available funding drives the choices that are made. As one interviewee 
said, “We have allowed the money to drive what we do.” In other cases, HA+C has 
sought funds to initiate or expand a program considered worthwhile. Because some 
of the funds obtained carry more restrictions than others, unrestricted funds are 
tapped whenever necessary to meet a need for which there is no other support. There 
are significant costs — and consequences — associated with operating in this way. 
Simply stated, it appears that decision making at HA+C has not been strategic. 
Strategic planning adds important elements to decision making. It allows goals to be 
set independent of external forces and provides a standard of measurement. It also 
converts values into action, and it allows limited resources to be committed in an 
orderly way. Many organizational experts recommend using organized strategies for 
making decisions. They believe that prudent analysis is the key to making good 
decisions, and that many decisions are made with too little information and too little 
thought. There are numerous tools for engaging in strategic or tactical decision- 
making 2 . These approaches are designed to compensate for the fact that people often 
believe they know the pluses and minuses of a decision before making it, but in 
actual practice, they make decisions before considering the evidence in an orderly 
way. Only after the decision is made do they seek evidence to support it. Deliberately 
and carefully considering the evidence on both (or all) sides before making a decision 
can have a major impact on the quality of those decisions. 


2 These have been as simple as pro/con T-charts or Edward De Bono’s PMI (pi us/minus/ interesting) 
process ( http://www.debonothinkingsvstems.com/home.htm ) for analyzing options. 


HOUSTON A+ CHALLENGE: AN ORGANIZATIONAL REVIEW 


10 


Figure 1. Executive Decision Making Screen For 
New Opportunities or New Initiatives 

1 . Alignment: is it true to the agency’s mission ? Is it 
consistent with and will it help us advance our 
strategic priorities ? 

2. Is the issue it will address really important? Is the 
activity or opportunity exciting? 

3. In what ways could the work add value to K-12 
practice or policy? In what ways can agency assets 
add value to the work? 

4. If it involves a partnership or collaboration, what 
are the pros and cons of this relationship? Is it 
likely to add value to our work and reputation and 
not undermine it? 

5. Does the activity come with funding? How much? 
Can we use it to obtain funding? From whom? 

How easily? Does the advantage of the opportunity 
outweigh the financial or time costs of taking it? 

6. Does the initiative put us into the political fray? 

Do we know enough about the politics to be able 
to handle ourselves competently? 

7. Does the funding source or partnership allow us to 
maintain our reputation for independence and 
impartiality? Can we anticipate and mitigate any 
challenges to these? 

8. Do we have the staff time to take on the work? 
Which staff will be impacted, and at what time of 
year? 

9. Does our staff have the skills, expertise, project 
management, and good judgment to take this new 
initiative on and execute it with high quality? How 
could the board or consultants help? 

10. Can the project complement other agency 
services/activities? Can it help meet other agency 
obligations and interests, generate visibility? 

1 1 . Can we execute this project well while maintaining 
credibility with our constituents, colleagues, and 
the board of directors? Is the value we might add 
by undertaking this initiative worth any strain and 
challenge it might bring? 

Source: EdSource, 
Mountain View, CA 


The first step in these strategic 
decision-making processes is a well- 
defined mission. (Figure 1 is an 
example of a tool used by an 
organization to make decisions about 
research topics on which to focus.) 

HA+C has many significant 
accomplishments to its credit. It has 
served as a catalyst for important 
discussions about education, and it 
has “planted enough seeds to change 
the way schools are run” — how they 
make decisions, plan instruction, use 
data, and solve problems. At present, 
however, HA+C does not have an 
organizational “culture of evidence” 
to support strategic decision making, 
a process that would enable HA+C to 
advance initiatives likely to have the 
most impact and sustain and scale-up 
important changes in schools and 
districts. 

It is also unclear just who makes 
decisions and how decisions are 
made. Who participates in decisions? 
What is the basis for making choices 
and setting priorities? When should 
the executive director and staff make 
decisions? When should consensus be 
sought among key stakeholders? 

How should the Board be involved 
in decision making about programs 
or projects? Some research on 
strategic decision-making 3 indicates 
that a decision isn’t necessarily a 


3 Nichols, F.W., Strategic Decision Making: Commitment to Strategic Action (Distance Consulting, 
2005). 
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choice among alternatives, but it is always a commitment to a particular course of 
action. Whether reached through bargaining or consensus or rounds of critique, a 
commitment must be made and then managed and supported. Finally, it must be 
communicated — clearly, coherently, and convincingly. 


Summary Points and Recommendations 

• HA+C has been driven largely by external forces: funders, school/district 
requests, school reform trends. 

• HA+C has no apparent process for examining the trade-offs in decisions, 
and it is unclear who participates in decision-making about program 
initiatives and how decisions and their rationales are communicated. 

• After ten years of work and with new staff leadership, HA+C is well 
positioned to adopt 

• a strategic-planning process to redefine its mission and develop decision- 
making and accountability procedures to guide its future work. 

• HA+C should develop or adopt a set of tools for establishing a clear process 
for making program decisions guided by a revitalized mission statement. 
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HA+C’s Portfolio of Programs and Initiatives 

HA+C has a large portfolio of programs and initiatives that reach a diverse 
constituency of students, teachers, administrators, and community members in the 
Houston area. Although HA+C program participants value the range of HA+C 
program offerings, supporting such an array of programs has posed problems for 
HA+C as an organization. 

The size and complexity of the HA+C portfolio appears to be the result of an organic 
and opportunistic approach to supporting school reform. One of HA+C’s strengths 
has been its responsiveness to the needs of those it serves, but that very responsiveness 
has resulted in an accumulation of projects and programs over a decade that are not 
necessarily the most targeted or relevant activities in the current reform context. 
HA+C’s broad mission has further encouraged the spread of its portfolio. 

Another potential reason for this ever-expanding portfolio is the lack of an “exit 
strategy” for HA+C programs. Identifying the conditions or events that should 
trigger a re-evaluation of a program’s utility or efficacy — and perhaps its 
termination — is crucial to building a strong, lasting reform support organization. 
Without such a process, the decision to continue an initiative is left solely in the 
hands of the project’s funder. 

One consequence of HA+C’s broad portfolio is that the philanthropic organizations 
supporting HA+C work may view the organization as willing to take on almost any 
project. If HA+C has embraced the many and varied opportunities presented by 
funders, those funders may have developed the perception that they can ask HA+C 
to fill almost any role in Houston school reform. They are interested in helping to 
improve education, and HA+C seems to meet their need for a good and fair steward 
of their funds. Whether this perception of HA+C’s role is accurate or not, HA+C 
and its funders would both benefit from a clearer picture of the organization’s goals 
and strengths. HA+C should craft its mission to convey its priorities to prospective 
funders, and those priorities should then be reflected in its portfolio of work. 
Thoughtfully choosing and supporting the most effective, relevant HA+C initiatives, 
weeding out those that are not the best use of HA+C resources, and carefully 
integrating new work will best serve local students and educators and ensure HA+C’s 
longevity. 
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As HA+C contemplates its focus and perhaps considers a more limited portfolio, it 
will be essential to develop a sound approach to dealing with initiatives that it may 
no longer support. The apparent popularity of HA+C programs among its clients is a 
testament to HA+C, but it also can present problems if HA+C decides to move in a 
different direction. 

The Regional High School Reform Network is an apt example of a situation where 
HA+C should take some care as the initiative comes to a close. There is some 
evidence of the Network’s success and popularity among participants. One 
participant who was interviewed commended the opportunity that it provided for 
“sharing and being at work with the neighboring districts — to really focus on certain 
issues and concerns that we have.” She appreciated using the protocols to “speak and 
voice what we’ve learned.” In particular, this and other interviewees acknowledged 
the value of developing a portfolio at the end of the five-year term: “We knew we had 
to do it; at first it seemed like pieces of paper in a binder, but what’s come out of it 
has been a great product for us on the campus. . .it’s not just something to turn in; we 
will continue with it so that the community can see what we are about — not just 
TAKS scores and what the newspaper says about us — now we will use it to show A+, 
the school board, and the community what we’re about.” Respondents frequently 
cited the portfolio product as one of the most valuable outcomes of their 
participation, because it was a clear demonstration to parents and the community of 
what is being accomplished at their school. 

As HA+C shifts away from this area of work, the successes and challenges should be 
documented as thoroughly as possible so that others can learn from the endeavor and 
where productive or useful strategies or models were developed, they can be 
developed and expanded in other schools. In general, there has not been a concerted 
effort to make the best use of the lessons that have been learned, to scale up 
successful models. The education field at large, as well as HA+C as an organization, 
will benefit from such analysis now and in the future. This kind of careful approach 
to winding down the program may also help minimize disappointment or 
resentment among its supporters. 

MPR’s assessment of HA+C operations includes helping the organization move into 
the next stage of its development, positioning it most effectively to serve Houston- 
area schools. HA+C has recently identified leadership as a focal concept for future 
work. Understanding the implications of a decision to define HA+C in this way will 
be crucial to the organization’s future success. What does it mean for HA+C to focus 
on leadership? How will this narrower scope of work affect existing programs that do 
not fall under the umbrella of leadership? How does this new direction affect clients 
and funders who view HA+C as a provider of services outside this scope of work? 



HOUSTON A+ CHALLENGE: AN ORGANIZATIONAL REVIEW 


14 


What will change in day-to-day operations for each staff member? Articulating a new 
direction is only the first step in redefining HA+C’s work; next comes the hard work 
of envisioning the resulting changes at every level of the organization. 

One fundamental aspect of redefining HA+C will be addressing the tension between 
being a nimble, responsive reform support organization and providing consistently 
high-quality programs. Program quality ultimately flows from the quality and skills 
of HA+C staff. Ensuring that staff have the capacity to meet new program 
imperatives as HA+C matures will be critically important. 


Summary Points and Recommendations 

• Program participants value the range of HA+C offerings, but supporting 
such a wide array of activities poses problems for HA+C as an organization. 

• Philanthropic organizations that support HA+C work may have developed 
the perception that the organization will take on almost any education 
reform work. 

• HA+C’s portfolio of programs and initiatives is so broad that many 
supporters are unaware of all that the organization does and of programs 
that could benefit them or their colleagues. 

• There has been little effort to capitalize on lessons learned: to scale-up 
successful models, communicate successful strategies, or transfer 
responsibility to sustain successful models or strategies. 

• HA+C’s leadership should examine the organization’s current portfolio of 
work in a systematic and strategic manner and concurrent with a 
clarification of the organization’s mission. 

• A facilitator who can guide the process of redefining the mission may be 
helpful, but this should be someone who can keep the process relatively 
simple and straightforward. 

• HA+C should establish a procedure for capitalizing on lessons learned from 
each project and creating an exit strategy for each project that is based on 
more than just the termination of funding. 
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Research and Accountability 


MPR’s examination of HA+C’s programmatic work and internal operations shows 
that outcome data gathered over the past ten years are not adequate to the 
organization’s current needs. HA+C staff are aware of the need to adopt more 
rigorous evaluation practices, but the organization may lack some of the necessary 
expertise to do so. Each HA+C program does not need the same level of evaluative 
rigor; but at the inception of each program, staff should consider the level and type 
of evaluation that will be appropriate and useful. HA+C needs to know what works, 
what doesn’t work, what should be supported in the future based on program 
efficacy, and what programs should be let go, either because they have been 
ineffective or because they lie outside HA+C’s mission. 

Some efforts to collect outcome data about HA+C programs are in evidence. 

Methods for measuring program success include counting the number of participants 
at HA+C events, satisfaction surveys for program attendees, and post-event 
discussions with partners about whether they successfully promoted the desired 
reform. HA+C staff have also kept observation logs during school visits and 
compared pre/post data for teachers participating in coaching groups. These data 
collection approaches give some sense of HA+C’s impact, but they do not permit 
rigorous and systematic analysis of the effects of the organization’s work. 

The original funding documents for the Annenberg Challenge for Beacon and 
Lamplighter Schools provide an example of what may have been the strongest 
evaluation effort found at HA+C. There was substantial support for evaluation. 

There was some evidence that the programs had positive effects on student 
achievement, but our review of the methods and results raised many unanswered 
questions. Based on information we collected, it seems likely that these evaluation 
results were not used to make strategic decisions about whether there should be 
continuing support for the Beacon and Lamplighter Schools or whether the models 
were curtailed prematurely. The first step in building a strong accountability system 
is identifying appropriate outcome measures, and the second is the use of proper 
methods. An important final step — one that needs careful guidance — is the 
interpretation and use of the resulting data. 

The High Schools for a New Society (HSNS) program and the related Regional 
High School Reform Network (RHSRN) are more in-depth examples of the 
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shortcomings of current efforts by HA+C to evaluate the impact of its work. Since 
the second year of its implementation, an external consulting team, researchers from 
the University of Texas, Austin, have evaluated the HSNS high school reform 
initiative. Meaningful, systematic analysis of this program’s success has been 
challenging. Some circumstances beyond the control of HA+C and the external 
evaluation team have impeded rigorous analysis of student outcomes. But the 
ongoing annual HSNS evaluation has been valuable as a strategic planning tool and 
represents a sustained effort to document changes resulting from HA+C investment. 

The Regional High School Reform Network, however, intended to scale-up the 
HSNS program to school districts surrounding HISD, has not had an external 
evaluation. HA+C never budgeted resources for an external evaluation of the 
RHSRN program, so there is no systematic documentation of the implementation 
process and outcome measures over the course of the initiative. While our review of 
evidence related to HA+C’s work in high school reform indicates that the 
organization has developed a specific approach to the work, and there have been 
pockets of success, the organization has not developed a consistent record of 
implementation, and very little evidence exists to corroborate or refute the validity of 
their approach. As a result, HA+C cannot use the results of this program to inform 
the field about the value of the approach or guide its own future investments. 

As HA+C moves toward a more focused mission and portfolio of programs, it will be 
critical to the organization’s success to initiate a clear, comprehensive system of 
accountability as a decision-making tool. Such a system will allow the organization to 
see where it currently stands and to gauge the value of its future work. Building a 
“culture of evidence” within the organization, which promotes the use of data as the 
guide for all decisions, will allow the organization regularly to revisit its mission and 
scope of work. 

However HA+C decides to design a comprehensive accountability system, it will 
benefit from creating a structured strategic decision-making process. The process of 
designing and implementing a system of accountability at HA+C will highlight an 
existing tension within the organization. HA+C has operated as an organization that 
is flexible and responsive to the needs and desires of clients and funders, but the 
introduction of a rigorous accountability system will necessarily reduce some of that 
flexibility. Tension may develop around how HA+C staff, clients, and funders 
understand the organization’s identity. Some may feel that it is abandoning its roots, 
while others may be more prepared to embrace a culture of evidence. Attention 
should be paid to how this tension manifests itself, so that concerns of stakeholders 
can be addressed as they emerge. 
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Summary Points and Recommendations 

• The lack of a strong accountability system is arguably the greatest weakness 
in HA+C’s operations. While there have been serious attempts to collect 
information about program results, they have often either been short- 
sighted or misdirected. In addition, many staff and Board members said 
that they were unaware or unclear about program evaluation results. 

• HA+C staff have collected feedback data consistently that have led to 
changes in format or content of programs, but there are almost no strong 
impact data that can be used to make strategic decisions. 

• HA+C must build its capacity to plan and carry out systematic evaluations 
and operate under an overall accountability system. This may mean adding 
a staff member to direct the effort or hiring staff with some evaluation 
expertise. The development of an accountability plan and internal 
evaluation capacity could also be done by an outside firm. 

• One or more specific HA+C projects or programs could be used as object 
lessons for developing clear accountability procedures. 
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Funding and Financial Structure 


A well-functioning non-profit operates with sound financial practices and sufficient 
and suitable funding to run its programs and organization. Funding aligns with 
organizational mission and goals, and as the organization develops, its leadership 
adjusts funding strategies to adapt to changing goals, priorities, and circumstances. In 
general, HA+ C has been in good financial standing, with sufficient funds available, a 
strong operating reserve, impressively low administrative costs, and a large percentage 
of unrestricted funds that allow flexibility in decision-making. Improvements in 
financial practices can support the larger goals of the organization. A regular strategic 
review of its finances deserves attention as the organization matures. For example, the 
Board and executive staff might consider the amount of reserves kept, the balance of 
income types (public, private, interest, and earned income funding), and guidelines 
for how unrestricted funds are spent. 

Our review of documents indicates that FIA+ C has been in the enviable position of 
having both breadth and depth of funding. More than 20 different organizations 
provide funds and one, the Annenberg Foundation, has supplied significant funding 
since FIA+C’s inception, including $4 million yearly over the past four years, 
supporting the bulk of program and overhead costs. Flowever, FLA+C should prepare 
for the changes likely to occur without the support, both financial and otherwise, of a 
keystone funder. Will clients’ interest in working with FIA+C change if associated 
grants are no longer offered to them? Flow will programs in progress be sustained? It 
may also be necessary to replace this key support with multiple funders offering 
smaller contributions. What kind of attention and stewardship would these 
foundations need to feel confident in making larger contributions later? How can 
they best be exposed to program successes? How can HA+C increase public funding 
or earned income to balance private foundation support? What are the program exit 
strategies if sufficient support cannot be found to supplant the Annenberg funding? 
Answers to these questions can help provide stability for the organization if a source 
of support is withdrawn. 

A balance of funds is important, but ensuring that funding supports the goals and 
mission of the organization is equally important. Staff and Board interviews indicated 
that restricted funds — those dedicated to particular programs — are sometimes 
accepted without considering how a new program will fit with the mission and goals 
of the organization. Such opportunistic funding can contribute to an unfocused 
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portfolio. A review of HA+C documents supports this view, showing that the 
organization has supported programs ranging from infant health to higher education 
over time. Strategic criteria determining when HA+C will accept restricted funding 
could help guide these decisions. Criteria might include whether the proposed 
program aligns with the agency’s mission, advances its strategic priorities, adds value 
to its portfolio, and can be well executed given current staff and expertise within the 
organization. 

HA+C documents also show that a good portion of funding is unrestricted — not 
earmarked for particular purposes — which permits flexibility in choosing whether to 
spend funds on particular programs, operations, or grants to schools and districts. 
This freedom is a luxury, but there is a risk that decisions will be made without 
overall goals in mind. How does HA+C make choices about the use of unrestricted 
funds? Making intentional and strategic decisions about unrestricted funds can 
strengthen the organization. 

In addition to operating programs, HA+C is also a grant maker. Of the 24 clients 
interviewed, approximately two-thirds had received funding from HA+C. According 
to those clients, HA+C has served effectively in this role. While it was beyond the 
scope of the interviews to determine if the grants themselves were effective, clients 
commented on the role of the grant maker in monitoring whether funds are applied 
as intended. One client described HA+C as having “very typical requirements 
compared to other granting organizations.” Some praised the flexibility with which 
the grants were awarded, allowing grantees to change how the grant was used if 
circumstances changed, for example, using funds for books instead of travel. Grantees 
received prompt responses to their requests for such changes. Many also found final 
reporting requirements useful — detailing the use of funds and successes and 
challenges encountered while implementing programs. Perhaps unsurprisingly, all of 
the grantees were enthusiastic about the funds, and several emphasized the need for 
HA+C to continue granting funds to support reform activities. 


Summary Points and Recommendations 

• Houston A+ Challenge is an organization with both breadth and depth of 
funding. 

• Houston A+ Challenge has been in good financial standing, with a high 
percentage of unrestricted funds that permit considerable flexibility in 
operation. 
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In preparation for inevitable changes, HA+C should evaluate the 
organization’s funding structure and systems to ensure that funding 
supports its priorities and mission. 

Executive staff and Board members should review specific aspects of the 
financial structure: the amount of reserves, balance of income types, and 
guidelines for accepting funds and allocating unrestricted funds. 

If grant-making activities continue, the associated accountability 
requirements should be strengthened. 
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Sustaining Programs and Their Impact 


To optimize their efforts, effective nonprofits consider how to sustain the impact of 
their programs generally and practices promoted within those programs specifically. 
For good programs to be sustained, they either need ongoing funding or a program 
design that eventually needs little or no funding. For good practices to be sustained, 
clients need support in implementing new skills in their school or district and in 
sharing their new practices with others. Ideally, good programs and practices are 
widely disseminated across as many schools and districts as possible. 

FIA+C apparently has given some consideration to making its programs sustainable. 
Staff and Board interviews revealed at least two successes in transferring program 
responsibilities to others. When 1TA+ C reduced school staff funding by 50 percent 
in Flouston Schools for a New Society, more than half of the participating schools 
filled the funding gap so they could retain their School Improvement Facilitators and 
coaches. In the New Visions Leadership Program, FIA+C transitioned program 
graduates to an alumni group in which participants themselves develop agendas and 
facilitate meetings. 

On the other hand, some staff and Board interviews suggested that some programs, 
such as the fine arts and dual immersion programs in the Beacon Schools, might have 
ended prematurely. Some said that when the primary funder ended its support, the 
program also ended. With more planning, FIA+C could sustain good programs after 
initial funds run out by developing an earned income stream, cultivating other 
funders before existing grants end, and using unrestricted funds, among other 
approaches. FIA+C could also benefit from systematizing its evaluation of program 
quality and decisions about whether to expand or sustain programs showing positive 
results. Flow can high-quality programs reach the most people? What tools and 
supports do graduates need to sustain and expand high-quality practices in their 
schools and districts? What funding is needed to replicate the program in other 
schools and districts? 

Obtaining new funding would likely be easier if FIA+C actively promoted its 
achievements more. Staff and Board members expressed concern that the 
organization is not making the most of its program successes. As one person put it, 
“The organization hasn’t taken a strong enough stance to say ‘this is our impact; 
here’s what’s been accomplished.’” Staff and Board members also felt that reforms 
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would be more likely to take hold in districts and schools if successes were more 
widely disseminated. 

In considering sustainability of programs, HA+C should gather data showing that 
programs are worthwhile investments to secure future funding, but it also needs to 
consider the following: 

• How can HA+C continue leveraging additional or new funds to sustain its 
work for particular programs? 

• How can HA+C measure the extent to which its programs have spread 
their impact more widely to participants’ schools and districts? 

• How can HA+C measure the extent to which programs have begun 
operating independent of HA+C funds or human resources? 

• What other indicators would be useful? 

• What processes should be put in place to monitor these indicators of 
sustainability? 

Client interviews revealed that current programs may not pay attention to how 
practices are sustained once participants leave the HA+C program. For example, 
when participants in the Critical Friends Group and teacher externship programs 
attempted to share what they had learned with peers at their schools, many realized 
that they were unsure how to transfer what they had learned, especially when faced 
with resistance to change. One participant said: “I’ve gone to several things with 
[HA+C] and my problem is always how to bring it back to my school and share it 
with other people.” Another participant explained: 

The meetings were very ‘intense’. . .we had a lot of homework, lengthy agendas at 
the meetings; we’d have to come back to our school and share our learning. 
Sometimes there was too much information, and it was hard to figure out how to 
bring it back to the school. 

Given this, HA+C may improve its sustained impact in schools and districts by 
paying more attention to how program participants bring their skills and knowledge 
back to their own work environments. As mentioned previously, staff and Board 
members felt that spreading the word about program successes could support 
program participants attempting to incorporate new practices into their work 
environments. 

Planning programs with sustainability in mind can also help strengthen the impact of 
HA+C offerings. For example, keeping districts and schools informed of program 
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goals and processes may make them more receptive when program graduates try to 
incorporate what they have learned into their school and district sites. Some clients 
suggested that collaboration between district and school leaders in implementing the 
Regional High School Network may have improved the program’s quality. Principals 
likely had a better understanding of their roles and responsibilities in overseeing their 
budget and the reform work, and district leaders were more committed to the 
reforms’ success. HA+C might investigate this issue further to determine how best to 
coordinate district and school personnel in future efforts, as well as consider how to 
encourage public entities to accept funding responsibility once they have seen proven 
program results. 

Summary Points and Recommendations 

• HA+C has had some success in transferring program and funding 
responsibilities to others (e.g., school districts) and in using strategies to 
sustain learning (e.g., alumni groups and reunions). 

• HA+C should initiate or plan programs with sustainability in mind, 
interfacing with school districts to ensure that participants returning to 
their sites do not meet undue resistance. 

• HA+C should consider assisting clients in developing their own strategies 
for sustaining an effort or transferring what they learn to their own schools 
or districts. 

• Before initial funds for a program expire, HA+C should develop an earned 
income stream, cultivate other funders, and plan for the use of unrestricted 
funds, among other approaches. 

• HA+C should promote its successes and develop plans to scale-up or 
propagate successful models. 
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Maintaining Quality Control 


Interviews with Board, staff, and program participants indicated that people associate 
HA+C with high quality. Respondents often mentioned that program content was 
powerful (using terms like “powerful,” “sophisticated,” and “cutting-edge”) and that 
delivery was well executed (“professional”). Some familiar with the organization’s 
grant management practices said that HA+C organization was a “good steward of 
funds” and “responsive” to inquiries and change requests. Some mentioned that 
HA+C is valued in the community as a way to maximize the amount of donor 
money that gets to school sites for their intended purpose. 

These same interviews, however, revealed some tension over how well HA+C could 
maintain that reputation in the future. Some questions may be directly connected to 
the organization’s broad scope of work. For example, some expressed concern about 
opportunities for staff to preserve and enhance their professional knowledge, given 
how many programs they were responsible for. HA+C also uses a number of external 
consultants to implement and coordinate some programs, and this can create 
additional challenges in overseeing quality and knowledge management. Some 
program participants worried that the quality of some programs may be deteriorating 
or their potential impact diminishing, as a result of expanding scope or narrowing 
networks. They feel that not enough new people or schools were participating in 
HA+C programs, impeding the development of a culture of reform. Some 
respondents also were concerned that not enough people were participating in 
follow-up training and, therefore, were losing sight of key principles. 

As the organization enters a new era, the quality of what HA+C provides as an 
intermediary may get more scrutiny from both its donors and program participants. 
Since the organization’s strong reputation is based in part on the perception that 
dollars go to schools, not overhead, but overhead costs may go up as more effort is 
expended to develop resources. HA+C will need to ensure that overhead costs are 
justifiable, that overhead tasks are done efficiently, and that work clearly aligns with 
specific purposes and generates measurable results. 

To maintain HA+C’s good reputation in the field of school reform, the organization 
needs to build systems to ensure that quality can be accurately measured and 
maintained. Quality control has many elements, two of which are people and 
programs. HA+C should consider the following questions: 
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1 . Do we have the right people doing the right things? 

2. Are programs in the field being implemented and replicated faithfully 
(conforming to the organization’s mission and vision) and consistently (across 
time and place)? 

HA+C must make sure it is doing what it does well and that those responsible for 
maintaining quality are supporting the agreed-upon goals and strategies. One client 
said that the external consultant facilitating his/her group regularly criticized HA+C 
to their group. Participant data also revealed several situations in which participant 
concerns about the quality of a program as implemented by an external consultant 
were addressed when HA+C staff became more directly involved in the oversight and 
implementation of the project. 

Another program participant mentioned that the opportunity to provide feedback at 
the end of network meetings was valuable because HA+C apparently listened to 
concerns and made changes. While collecting participant feedback is a critical aspect 
of an effective quality-control system, it is unclear if anecdotes such as this are the 
norm. Our document review showed that HA+C does collect ample participant 
feedback data, but it was not clear if these data are consistently reviewed or used. It 
was unclear whether HA+C takes specific steps to address concerns or to follow up 
with participants to see if the intent to act on what they learned is eventually 
translated into changes in practices. It seems that current staff workloads preclude 
systematic use of such practices. Using reliable measurement methods, however, will 
help HA+C determine what it does best and thereby what should be the scope of its 
work. Careful measurement can also help maintain program quality and consistency 
if HA+C’s leaders systematically use data to drive their decision making about staff 
and resource allocation. 


Summary Points and Recommendations 

• HA+C’s work is consistently seen as high quality. 

• Maintaining high quality requires constant monitoring and using feedback 
and impact data to make course corrections as needed to keep HA+C in 
line with its mission and to ensure desired outcomes. 
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Communication 


Across the various groups of respondents we interviewed, we saw a pattern of an 
inadequate level of communication to convey the mission and scope of the 
organization as well as strategies for sustaining the work and documenting its 
accomplishments. One effect of HA+C’s large portfolio of work is that clients often 
indicated that they unfamiliar with HA+C work outside of the programs in which 
they participated. Most of these clients know that HA+C is involved in many 
activities, but they did not feel informed about the nature of all of these programs. 
Several clients felt that knowing more about what HA+C offers would be beneficial, 
as they could help identify other potential participants. One program participant 
described her HA+C program as one of the district’s “best-kept secrets,” and wished 
that HA+C would do more to promote the program to other educators and the 
community. A clear, well-publicized menu of HA+C programs would almost 
certainly lead to broader and more effective dissemination of information about its 
valuable work for Houston-area schools. 

Staff members reflected less than sufficient awareness of how decisions were made 
about the work the organization would undertake, how the work of various programs 
should inter-relate, and how to ensure that their work was sustained. Some Board 
members were not well versed in what HA+C does, and some mentioned that the 
community was not adequately informed about the work of the organization and, 
therefore, might not be inclined to support it. It was evident that Board and staff did 
not interact regularly. Some organizations routinely bring Board and staff together in 
retreats to plan and strategize. Such efforts can enhance communication, while also 
promoting synergy and cohesion among those most involved in carrying out HA+C’s 
mission. 


Summary Points and Recommendations 

• The organization evidences a lack of effective communication strategies at 
all levels: between staff and Board and between the organization and its 
constituents/ community. 

• HA+C should develop a strong communication plan that outlines specific 
strategies for ensuring effective communication and a set of topics about 
which it must communicate effectively. 
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The Staff and Board of Directors 


As HA+C moves forward and refines its portfolio of work, it should consider the 
range and level of staff capacity. With a more defined mission and decisions made 
about HA+C’s direction over the next ten years, appropriate staff must be in place — 
or provided the professional development needed — to carry out the mission. The 
staff themselves, as well as others interviewed, noted that current staff do not have 
the necessary content expertise to engage in some HA+C activities if the organization 
moves in new directions. 

HA+C staff generally work hard at their jobs and are good at specific aspects of their 
particular program responsibilities. They seem to feel successful, but they express 
misgivings about how their work fits within the scope and direction of the 
organization and how it can have sustained impact. Some Board members were 
unclear about what exactly the staff does. What was most frequently heard — even 
from the staff themselves — was that they act as facilitators or conveners. They are 
good at bringing people together and facilitating conversations on issues of interest 
and/or concern. Because they are not content experts, for the most part, however, 
they are often in situations where they cannot necessarily “lead change.” Using the 
horse-and-cart analogy, staff seem effective at being the cart, but they may want or 
need to be the horse more often, to be truly strategic and provide the leadership 
essential to promoting reform. These staff concerns are related to HA+C’s unclear 
mission and its past unwillingness to be strategic. If an organization’s mission and 
desired outcomes are not specific, it is hard to know what staff knowledge and skills 
are needed to fulfill that mission. 

Successful organizations can define with precision who they are and who is 
responsible for what — guiding the mission, delivering services, monitoring and 
collecting data on implementation, finding support for the mission, networking, 
facilitating, and so on. This often involves difficult decisions that organizations can 
be reluctant to make. It can also be an opportunity, however, to develop existing 
capacity or help staff re-tool to take on new responsibilities as the organization moves 
in new directions. 

While HA+C must determine if the capacity and balance among its staff are suited to 
its mission and goals, attention also should be paid to enhancing the Board’s 
understanding of the organization’s work and how each member participates in that 
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work. Some Board members apparently are deeply involved in HA+C, while others 
are much less informed and less engaged. This is not unusual for a Board, but efforts 
should be made to ensure that the Board can support the organization in optimal 
ways. Boards generally do not want to be inundated with information about program 
operations, but they should receive sufficient information to allow them to maintain 
a level of engagement that is comfortable, appropriate, and productive for the 
organization. 


Summary Points and Recommendations 

• HA+C has a committed staff willing to work hard work and establish a 
strong presence and good reputation for the organization. They are well 
informed about the operation of schools and districts, and this has enabled 
them to forge solid connections with their constituents. 

• HA+C is likewise fortunate to have a Board of Directors that includes 
accomplished business persons, educators, and professionals. These 
individuals have provided invaluable conceptual help, social capital, and 
fund-raising support. Some are strongly involved in defining the mission of 
the organization, but others are less engaged and less informed. 

• HA+C must define the human capital needed to move the organization in 
the direction it chooses through strategic planning. The organization must 
have the right staff to meet its goals. 

• HA+C should also review the mix of Board members and, at a minimum, 
strive to ensure that the Board is as informed and engaged as needed to 
provide optimum help to the organization. 

• HA+C leadership may want to seek information about documented “best 
practices” for organizational staff and Boards to help the organization 
consider how it can best conduct its work and fulfill its mission. 
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Conclusion 


It is certainly clear from this review that Houston A+ Challenge has established a 
firm presence in the Houston education arena as a strong, responsive organization 
that is making significant contributions to the improvement of schools and districts. 
It is known as an organization that provides access to cutting-edge research, affords 
opportunities for teachers and administrators to participate in high-quality 
professional development events or networking activities that have resulted in 
meaningful change in the schools and districts from which participants have come. It 
has also been a good steward of funds that have been distributed to schools and 
districts and used to benefit students throughout the Houston area by improving 
teacher practice, providing learning opportunities outside of schools, building 
successful models of instruction, and personalizing learning experiences for both 
students and educational professionals. There is a clearly a great deal of value that 
they have added to Houston area education. 

Houston A+ Challenge now finds itself at a crossroads, and the time is propitious for 
the organization to address significant operational issues that have been detailed in 
this report. These issues center around the crafting of a clear, defined mission 
statement and process of making decisions that are couched in a definitive 
accountability system that will provide the basis for moving forward. With the 
elements in place that we have recommended, we have no doubt that Houston A+ 
Challenge will embark on the next ten years of its journey with the promise of 
making an even greater contribution to the Houston-area schools and the children 
within them. 
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Appendix 

Methods 

MPR began this organizational review by observing the Reforming Schools Summer 
Institute (RSSI) and training for the new math coaches during the summer of 2007. 
In early October 2007, six researchers from MPR conducted a week-long site visit to 
Houston, during which we participated in several events, mostly presentations or 
work sessions involving visiting expert Michael Fullan. We also participated in a staff 
meeting, conducted one-on-one interviews with staff and Board members, and 
reviewed documents and data from the HA+C files. We later analyzed and 
synthesized these data and conducted additional phone interviews with staff, Board 
members, and clients. 


Participation in Events 

While on site in October, MPR staff participated in several events: three days of the 
RSSI; two staff meetings and the summer training for the new middle school math 
coaches; the training for math coaches by Education Development Corporation 
(EDC); and the initial Critical Friends Group training session for Goose Creek 
Independent School District. Staff also participated in several sessions led by Michael 
Fullan: 1) a Speaker-Series session with Leadership Academy members; 2) a meeting 
with Board members and the Leadership Academy Design Team, and 3) a 
community session. 


Review of Documents and Data 

MPR staff reviewed electronic and paper ELA+C files documenting the organization’s 
work over the last ten years, focusing on specific materials related to the 
organization’s predominant and most recent activities and initiatives. We also 
obtained files from Program Coordinators on both recently completed and current 
projects. In our on-site review, we examined documents showing 1) the specific work 
and actions of organization staff and affiliates; 2) the immediate impact of HA+C 
work (e.g., numbers attending meetings); 3) costs associated with particular activities 
and initiatives; 4) program participants’ response to activities and initiatives; and 5) 
evidence of program impact. We requested and received additional documents 
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related to specific initiatives from individual Program Coordinators and additional 
financial documents from the Finance Director. We examined these documents for 
further information on the design and management of FLA+C work as well as its 
associated outcomes. 

Interviews with Staff and Board Members 

We conducted interviews of approximately one hour with all FLA+C staff members 
and Board members. A semi-structured interview protocol was developed and used, 
and interviews also were digitally recorded for use in clarifying notes. We examined 
data from these interviews for patterns, themes, and useful recommendations, and 
then we synthesized data from interviews according to themes that emerged from a 
series of debriefing discussions among research staff. 


Respondents 

Number 

HA+C Staff 

Michele Pola (former Director) 

Scott Van Beck (current Director) 

Suzanne Sutherland 

Tim Martindell 

Catherine Reed 

Alejandro Morua 

Mike Webster 

Brandi Allen 

Angela Prince 

9 

FLA+C Board Members 

Susan Bischoff 
Jonathan Day 
Ann Friedman 
Joe Foster 
Roberto Gonzalez 
tfarry Reasoner 
J. Victor Samuels 
Andrea White 

8 

Participants/Clients 

24 


Phone Interviews with Participants/Clients 

To learn about the response of program participants or clients to the work of FLA+C, 
we obtained lists of participants in the various programs sponsored by FLA+C. We 
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sorted participants by district, role, and program areas (e.g., leadership, teaching and 
learning, community, and high schools) in which they participated. We then selected 
all who had participated in multiple programs and added to them a proportional 
sample of participants representing the different program areas and school districts. 
From these, we randomly selected a sample of 36 to contact for interviews, with a 
goal of conducting 24. Using a semi-structured protocol, we conducted 
approximately 30-minute telephone interviews with these individuals, asking them 
about their experiences with HA+C (e.g., their perception of their mission, reactions 
to events or initiatives in which they participated, and the impact of their 
participation on their work.) We also asked if they had suggestions or 
recommendations for the organization as it moves forward. 
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Interview Protocols 


Houston A+ Challenge 


Questions for Board members: 

1. How long have you been a Board member? 

2. How did you come to be involved with the Board? 

3. What do you see as the contribution you can make? (i.e., why did you want to 
participate?) 

4. What do you see as the mission of HA+C? 

5. What do you think have been its major contributions to education in the Houston 
area? 

6. Where do you think it should go from here? (How do you think it should refine 
its mission or strategies for accomplishing the mission?) 

7. What background/experience/characteristics do you think the new ED should 
have? (W hat would you be looking for?) 

8. Is there anything we haven't asked you that you'd I ike to tel I us about the work of 
HA+C? 

• How would you describe the primary "problem" in districts that H AC is 
addressing? 

• Do you feel that H AC is adequately addressing that problem? In what ways is 
HAC excelling? 

• How do you think HAC could be better serving these districts? 

• We have categorized the H A C work in four groups: 

1. School networks 

2. M odel initiatives 

3. Leadership 

4. Policy 

How well do you think HAC is implementing programs in each of these 
areas? 

• Asa board member, how do you ensure that HAC is achieving its goals and 
staying in line with its mission? What indicators do you monitor? 



How does HAC and its board determine which large grants to accept when they 
are offered? Do you have set criteria for assessing whether to accept them? 



Questions for Staff: 


M PR isconducting exploratory interviews with key HA+C staff and several stakeholders this week in preparation to 
conduct an organizational evaluation of HA +C. This interview is not part of the actual evaluation but will provide 
information that will allow M PR to design that evaluation. Please know that neither this interview nor any 
subsequent information gathered as part of the evaluation process is being used to evaluate you or your work 
individually. Y our responses are completely confidential. 

We are hoping to gather information about the specific functions of the organization and its staff as HA+C works 
toward meeting its goals and how those goals and functions have evolved over time. We are also interested in 
knowing what staff and stakeholders imagine for the organization in the future and what you believe would be 
needed to get there. 

Name of interviewee - 

Job title - 

If applicable: 

Current initiatives directly responsible for: 


Past initiatives directly responsible for: 

1. W hat is your background/experience? 


2. How long have you been with the organization? How would you characterize changes that have occurred in 
the organization? 


3. (W e know your title), but how does that translate into your overall responsibilities and your daily tasks? 


4. As far as your responsibilities to specific initiatives, how and by whom are those responsibilities 
determined? 


5. W hat do you see as the mission for HA +C? (In your perception, what is the key function that HA +C serves 
currently? Has that changed overtime? How so? Before coming to work for HA+C, did you have any prior 
contact with the organization? I n what capacity? ) 


6. W hat do you think have been its major accomplishments? (probe on those regarding specific details of 
what it was, how it came about, factors that contributed, who had major responsibility, how they knew it 
was a success) 


7. What have been its (H AC's) biggest challenges, problems (barriers, shortcomings, dysfunctional areas)? 


8 . 


Focusing on the 


initiative, what have been that program’s biggest accomplishments? 



9. What have been its (the initiative's) biggest challenges, problems (barriers, shortcomings, dysfunctional 
areas)? 


10. How do you determine whether or not a program is going well? Do coordinators communicate this 

information across programs - how? How do you determine (as an organization) what strategies, programs, 
initiatives you will add to the work of HA +C? 


11. W hat types of opportunities does the organization provide for staff to build content knowledge, i.e. 
knowledge that could be useful to schools and the community? 


Projective 

12. I magine it is 2012. What would you like to be able to say about how H A+C is working to improve 

education in Houston areas schools? Where would you like the organization to be in 5 years— what will the 
mission, programs or major strategies be? How will it be seen as an organization? How will people 
describe it? W hat staff will it have (numbers, expertise, roles) 


13. W hat are some of the key things that would need to happen for that vision to be realized? 


14. W hat would you want from the organization's leaders in order to help the staff achieve that vision? (W hat 
changes (if any) do you think need to occur in the mission, programs, or daily operations? 


15. What background/experience/characteristics do you think the new ED should have? (What would you be 
looking for?) Do you see the need for other staff, other areas of expertise? 


16. I magine for a moment that HA+C didn’t exist... 

W hat, if anything, would be more difficult in terms of improving schools and student achievement in 
Houston? 

W hat things do you provide that would be missed so much, someone would try to provide them? 

Are any HA+C influenced practices that are happening in the schools now could sustain themselves 
without your input? 



Interview Questions 

for Participants in Houston A + Programs 


Background 

1. Please describe briefly your background in education, including your current as well as former 
positions. 

2. H ow long have you been working in education? 

Impressions of A+ 

3. How would you describe the overall purpose and goals of Houston A + Challenge (i.e., if 
someone asked you what it is or what it does, how would you answer)? 

4. It's my understanding that you have participated in the following events/activities sponsored by 

Houston A +: Is that correct? (If not, add correct information: 


5. Could you tell me overall what your experience was with these activities? (How would you 
assess the quality of them— format, content, presenter, logistics? What general reactions did 
you have to the program (or event or activity)? Did you have different experiences with 
different activities (if participated in more than one)? (If activity involved sequence of 
meetings, ask if person was able to attend majority of meetings.) 

6. Have you or your school received a grantor direct funding from Houston A +? How were the 
funds managed? What kind of monitoring or reporting did you need to do? (Was that helpful, 
burdensome?) 


I mpact/Perfor mance 

7. Can you give me examples of knowledge or skills that you gained as a result of your 
participation in the Houston A +... (activity, program, event)? 


8. W hat aspects of the A + program have been most hel pful to you? 


9. What would you like A + to do differently in the future? What would make their services more 
useful to you? (e.g., more frequent/different offerings, more/different content or focus, more 
1:1 support, better continuity of content from one meeting to the next, etc.) 


10. Do you have any suggestions or recommendations about how Houston A + could or should play 
a role in improving education in the Houston area in the future? 



